Evidence profiling

ABSTRACT

Evidence profiling, in one aspect, may receive a candidate answer and supporting pieces of evidence. An evidence profile may be generated, the evidence profile communicating a degree to which the evidence supports the candidate answer as being correct. The evidence profile may provide dimensions of evidence, and each dimension may support or refute the candidate answer as being correct.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/386,072, filed on Sep. 24, 2010, which is incorporated by referenceherein in its entirety.

FIELD OF THE INVENTION

The present application relates generally to computers, and computerapplications, and more particularly to artificial intelligence andnatural language processing.

BACKGROUND

In systems that produce or evaluate candidate answers, also referred toherein as hypotheses (statements that are posed as true), it isdifficult for end users to understand the origin or evaluation of aparticular hypothesis. It is also difficult for users to comparemultiple hypotheses and understand why the system prefers one hypothesisover another. Current systems fail to organize evidence for a hypothesisinto a semantically meaningful, intuitive, and comprehensive view forthe user so that the user can easily understand the system's evaluationof a hypothesis.

BRIEF SUMMARY

A method for evidence profiling, in one aspect, may include receiving acandidate answer and supporting pieces of evidence. The method may alsoinclude generating an evidence profile communicating a degree to whichthe evidence supports the candidate answer as being correct, wherein theevidence profile provides dimensions of evidence and each dimension maysupport or refute the candidate answer as being correct.

A system for evidence profiling, in one aspect, may include a moduleoperable to execute on the processor and receive a candidate answer andsupporting pieces of evidence. The module may be further operable togenerate an evidence profile communicating a degree to which theevidence supports the candidate answer as being correct, wherein theevidence profile provides dimensions of evidence and each dimension maysupport or refute the candidate answer as being correct.

The system may also include a visualization logic module operable toenable visualization of the evidence profile such that each dimension ofevidence can be selected and a weight associated with each dimensionaltered.

A computer readable storage medium storing a program of instructionsexecutable by a machine to perform one or more methods described hereinalso may be provided.

Further features as well as the structure and operation of variousembodiments are described in detail below with reference to theaccompanying drawings. In the drawings, like reference numbers indicateidentical or functionally similar elements.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a diagram illustrating a process of drilling down from rankedhypotheses through evidence profiles to original sources in oneembodiment of the present disclosure.

FIG. 2 shows an example answer produced by a methodology of the presentdisclosure in one embodiment as a candidate for completing a hypothesis.

FIG. 3 shows examples of evidence dimensions for a single hypothesis inone embodiment of the present disclosure.

FIG. 4 shows another example of visualization in one embodiment of thepresent disclosure.

FIG. 5 shows visualization for hierarchically drilling-down into one ofthe evidence dimensions in one embodiment of the present disclosure.

FIG. 6 shows another example of evidence profile visualization in oneembodiment of the present disclosure.

FIG. 7 shows an example of another dimension that played part indetermining an answer in one embodiment of the present disclosure.

FIG. 8 is a flow diagram showing a method of evidence profiling in oneembodiment of the present disclosure.

FIG. 9 illustrates generating an evidence profile in one embodiment ofthe present disclosure.

FIGS. 10A-10C illustrate an example QA system, for instance, which mayprovide a statement of hypothesis.

FIG. 11 illustrates a schematic of an example computer or processingsystem that may implement the evidence profiling in one embodiment ofthe present disclosure.

DETAILED DESCRIPTION

A question answering system takes a natural language question as inputand produces candidate answers as hypotheses. These candidate answersare scored and ranked in some relative order, but the justification forthis scoring and ranking up to now has been hidden from the end user. Asa non-limiting example, the methodologies described herein may beutilized in question answering (QA) systems such as those described inU.S. patent application Ser. No. 12/152,411 and U.S. patent applicationSer. No. 12/126,642, which are incorporated herein by reference in theirentirety. The present invention may be used in conjunction with anysystem that produces and/or evaluates hypotheses.

A QA system may take as input a question expressed in natural humanlanguage and, in return, produce a precise answer or ranked list ofanswers to the question along with a confidence score for each answer.Here each candidate answer is a hypothesis, and the QA system mayevaluate each candidate answer based on the originally input question.The candidate answer can be obtained from a corpus of documents anddata, both structure and unstructured. A variety of analytics are usedto analyze the question, obtain candidate answers, analyze supportingevidence for each candidate answer, generate feature scores for eachcandidate answer, and combine individual feature scores into a finalconfidence score for each candidate answer.

Given just the final ranked list of candidate answers and confidencescores and/or ranking, the user may have no insight into why thecandidate answers received their raw confidence scores or relativeranks. The user may benefit from more detailed information on why thesystem prefers one hypothesis over another. This information is usefulto a range of users. For the end user of the QA system, this detailedinformation may provide more background knowledge about the answergenerated by the system and increase the user's confidence in thequality of the result. For the system developer, a detailed view thatexplains how the confidence scores for each hypothesis are generated maybe an invaluable tool for debugging and tuning the system.

The methodology of the present disclosure in one embodiment may providean explanation of how an automatic question answering system produced afinal set of ranked candidate answers in response to a natural languagequestion. In the present disclosure, a hypothesis may be a candidateanswer to a natural language question automatically obtained by aquestion answer system. The system may store evidence to support thehypothesis obtained from a corpus of unstructured, semi-structured, orstructured information. The methodology of the present disclosure in oneembodiment automatically assigns weights or scores to the evidence basedon analytics that analyze the original question, hypotheses, andevidence. A large number of fine-grained analytics may produce hundredsof scores for a hypothesis, which can then be hierarchically organizedinto higher-level evidence profile dimensions, providing users with amore practical and easy to comprehend view of why one hypothesis scoreshigher than another. The methodology of the present disclosure in oneembodiment may automatically generate the hypotheses, the evidence, andthe evidence profile.

Existing methodologies do not consider arbitrary, fine-grained evidencethat can be organized into hierarchical evidence dimensions, and doesnot in any way provide measures, comparisons, or visualizations thatindicate to the end user which evidence dimensions cause one hypothesisto be ranked higher than another hypothesis and how individual pieces ofevidence contribute to each hypothesis. In contrast, the methodology ofthe present disclosure in one embodiment may provide a how each evidencedimension contributes to the score of a hypothesis, and how differenthypotheses compare to each other along arbitrary evidence dimensions.

The present disclosure in one embodiment provides a system and methodfor presenting this detailed information, referred to herein as an“evidence profile.” The evidence profile explains why and how ahypothesis is supported by the content. The evidence profile allowsusers to compare multiple hypotheses and understand why one hypothesisis preferred over another. The evidence profile also shows how theunderlying content used to produce and evaluate the hypothesis supportsor refutes the hypothesis along a number of different evidencedimensions.

A methodology of the present disclosure in one embodiment may organizethe underlying features and scores used to evaluate hypotheses intoevidence dimensions. An evidence dimension is a semantically meaningfulcollection of underlying feature scores, such as geographic, taxonomic,temporal, popularity, and others. Evidence dimensions are hierarchicaland may be expanded into more detailed dimensions or collapsed intohigher level dimensions. In one embodiment of the present disclosure,all of the evidence dimensions together form an evidence profile for thehypothesis.

FIG. 1 shows a chart explaining how candidate answers or hypotheses 104may have evidence profiles 102 providing meaningful information aboutindividual pieces of evidence 110. Evidence profiles 102 in oneembodiment of the present disclosure act as a central component of anoverall result exploration process. For instance, an exploration maystart with the ranked hypotheses 104, or candidate answers. Tounderstand why a particular hypothesis received its score, the user maythen navigate to the evidence profile 102 for that hypothesis. Theevidence profile 102 in one embodiment reveals the various dimensions ofevidence supporting one or more hypotheses with given confidence levels106 and leads the user to a full provenance chain of associated piecesof evidence 110 and their scores. Examples of evidence dimensions (e.g.,shown at 108) may include items such as: Taxonomic, Geospatial(location), Temporal, Source Reliability, Gender, Name Consistency,Relational, Passage Support, Theory Consistency, and/or others. Thedimensions may depend on the types of algorithms that were used togather and score evidence. Once exploring an evidence dimension the usercan navigate to the source evidence 110, such as the actual passages,documents, or database facts that were used to produce the score on thatdimension.

Evidence profiles 102 may be compared for competing hypotheses. As anexample, FIG. 1 shows four possible answers to a question represented byhypotheses ranked by a system's assigned confidence scores 106. Two areselected, Saint Paul 112 and South Bend 114, for further explorationthrough evidence profiles 102. Next, the system may display acomparative evidence profile showing a select set of evidence dimensionsfor each answer (e.g., 108). Drilling down further in any one of thedimensions in the evidence profile produces the original source contentused to produce the corresponding confidences.

FIG. 2 shows a comparison of evidence profiles for candidate answers(i.e. hypotheses) Argentina and Bolivia. As a non-limiting example, bothArgentina and Bolivia were produced by a question answering system basedon the following input:

-   -   Chile shares its longest land border with X.

If the ideal source is accessible or available, this is a question thatcan be answered with complete certainty, but for the sake ofdemonstration it is assumed that the available content did not includethe ideal source. FIG. 2 shows that Argentina is favored as the correcthypothesis over Bolivia in the dimensions of location 202, passagesupport 204 and source reliability 208. Whereas Bolivia is favored asthe correct hypothesis over Argentina in the dimension of popularity206. A question answering system could adapt to stronger evidencedimensions through training. With stronger and more precise geospatialcontent and reasoning, for example, the system would learn to weightgeospatial evidence higher and the weighted bar for that dimension mighthave alone outweigh other evidence.

A methodology of the present disclosure in one embodiment may beimplemented by recording the feature scores assigned to a givenhypothesis by the various analytics that score hypotheses, grouping thefeatures into semantically meaningful evidence dimensions, andvisualizing the contributions along each evidence dimension to show theoverall feature profile for the hypothesis.

The evidence profile may be visualized in a variety of different ways. Anumber of visualizations of the evidence profile may provide a range ofuser types with a wide variety of techniques for understanding thejustification for a particular hypothesis, comparing hypotheses,debugging system behavior, and others.

FIG. 3 shows examples of evidence dimensions for a single hypothesis. InFIG. 3, the question posed is “You'll find Bethel College & Seminary inthis “holy” Minnesota city”. In the visualization shown in FIG. 3, theevidence profile for a single hypothesis, Saint Paul, is given, showingthe relative contributions, both supporting and opposing, along eachevidence dimension for the hypothesis.

FIG. 4 shows another example of visualization showing a comparison ofevidence profiles for two hypotheses. Using a comparative evidenceweight plot, the evidence profiles for two competing hypotheses arecompared and differences along each evidence dimension are highlighted.For example, an evidence weight plot is one example of visualization forshowing the relative contributions of each evidence dimension for agiven hypothesis. The evidence weight plot can be used to visualize thecomparison of two hypotheses by showing only those dimensions thatprovide different levels of support for the two answers, and mayoptionally show the relative differences.

FIG. 5 shows yet another example of visualization showing a comparisonof evidence profiles for two hypotheses. FIG. 5 shows visualization forhierarchically drilling-down into one of the evidence dimensions. InFIG. 5, one of the dimensions may be drilled down into, and the detailsof more fine-grained evidence dimensions may be viewed. For example,“geospatial” dimension may be include “SpatialDistance”,“SpatialDistance-Std” and SpatialRelationSat-Std” fine-grained evidencedimensions. In another embodiment, not shown, the evidence dimensionscan be drilled down even further to provide individual pieces ofevidence and the associated contribution to each evidence dimension.

FIG. 6 shows another example of evidence profile visualization thatanswers the question for finding a city in Minnesota in which BethelCollege and Seminary is located. The bars show the contribution of eachdimension to the final confidence score for the answers provided. Whilethere is a Bethel College in both Saint Paul and South Bend, the correctanswer is Saint Paul because it is in Minnesota. A user might not knowthat Saint Paul is in Minnesota and that South Bend is not. However, auser would be able to tell from the question that location might be adimension that would need to be weighed heavily to determining thecorrect answer. If a user was provided with the hypotheses of both SaintPaul and South Bend, the user could use the evidence profile visualizedin FIG. 6 and understand the certain dimensions, like location, shouldbe weighed higher and other dimensions, like popularity, should beweighed lower. As such, the user could alter the weight associated withthe evidence dimensions in order to arrive at the correct answer. Theevidence profile showing each dimension and contribution to theevaluation of each hypothesis allows the user to make this change andobtain a correct answer. If this functionality were not available to aquestion answer system then an incorrect answer may be provided and theuser would have no ability to discern whether the system wasappropriately weighing the evidence, or confidently rely on the answerprovided.

FIG. 7 shows an example of another dimension that played part indetermining the final result (answer). For instance, puns may be anotherdimension considered for arriving at the answer. Humans may answer thequestion based on the pun since Saint Paul implies a “holy” city. A punscorer may be added as another dimension that will discover and scorepun-like relationships. Again, a user may be able to determine that thequestion is a pun and therefore more weight given to the pun relation702.

The evidence profiling system and methodology in the present disclosuremay be incorporated directly into a question answering system or be aseparate module. For example, a question answering system may take aquestion, and from a corpus of data, generate candidate answers orhypotheses and associate supporting evidence. These hypotheses andassociate supporting evidence could then be sent to a separate evidenceprofiling module that receives the hypothesis as input and generates oneor more evidence profiles communicating the degree to which thesupporting evidence supports or refutes the hypothesis as being correct.Evidence profiles may contain one or more different evidence dimensionsand provide “scores” or values for each evidence dimension such that thescores for each dimension are comparable across different hypotheses.

FIG. 8 is a flow diagram showing a method of evidence profiling in oneembodiment of the present disclosure. At 802, a statement such as ahypothesis or answer to a question is received. Additionally, supportingevidence is received associated with each hypothesis. Examples ofsupporting evidence may include passages, documents, and/or entities ina database. At 804, an evidence profile is generated that describesvarious supporting evidence grouped into various dimensions and given acomposite score contributing to a confidence score for each hypothesis.At 806, one or more visualizations may be generated to present theevidence profile.

FIG. 9 illustrates generating an evidence profile in one embodiment ofthe present disclosure. At 902, dimensions of evidence supporting thehypothesis are determined. The dimensions may depend on the types ofalgorithms that were used to gather and score evidence. For example, ataxonomy of dimensions of evidence may be created that organizesalgorithms used to score candidate answers into different dimensions ofevidence. The taxonomy may be hierarchical, such that a given dimensionof evidence can be specialized into a number of sub dimensions whereeach sub dimension of evidence represents a more specific kind ofevidence under the parent dimension. The evidence profile can then beviewed at different granularities, starting with the top level, mostcoarse dimensions, or drilling down into the more specific subdimensions of any particular dimension. The taxonomy is created suchthat a person using the evidence profiling system would be able tounderstand the meaning for each group or dimension. Based on thistaxonomy and the associated algorithms, pieces of supporting evidenceare categorized into certain dimensions within the taxonomy.

Examples of dimensions of evidence may include temporal, location,passage support, classification, popularity, source reliability,predicate plausibility, document support, hidden link, and/or others.Temporal dimension refers to events and people that happen duringparticular times and have likely life extents. Location dimension refersplaces, places located in or near other places, and to events that mayhappen in particular places. Passage Support dimension refers topassages that relate key entities to a candidate answer. Passage Supportdimension may further include Shallow Evidence referring to passagesuperficially aligned with question text, and Deep Evidence in whichcandidate answer is understood to be in the right relationship with keyentities based on logical analysis of passages. Classification dimensionspecifies that candidate answer should be of the right type (e.g.,Woman, King, Disease, Invention). Popularity dimension specifies thatAnswer is popularly associated with parts of question. SourceReliability dimension specifies that Sources supporting answer arelearned to be reliable. Predicate Plausibility dimension specifies thatthe answer is reasonable based on the role it plays in key relations.Document Support dimension specifies that the Document appears todiscuss fact in context of answer. Hidden Link dimension specifies thatcandidate answer and question entities share common thread.

At 904, the scores of each of the dimensions may be obtained. Dependingon how the dimension was created, this can be done in a number of ways.Algorithms relating to the dimension can be aggregated for example. Theaggregation can be a summation, averaging, weighted averaging, or anyother mathematical or statistical algorithm that aggregates multiplevalues into a single value.

At 906, different forms of visualization may be provided based on thedetermined dimensions and associated scores. For example, a simplelisting of evidence dimensions for a hypothesis may be listed along withcorresponding scores such as shown in FIG. 3. Or, one or more scores forone dimension may be compared to one or more scores for anotherdimension by listing scores and/or a bar chart displaying the differencesuch as shown in FIGS. 4 and 5. A preferred method of displayingdimensions is to provide a bar chart such as shown in FIGS. 6 and 7 thatallows a user to quickly and easily determine the relative contributionof each dimension to the confidence score for each hypothesis.

In one embodiment of the present disclosure, evidence dimensions may beaggregated into a hierarchy. For example, the evidence profile maycontain base dimensions and aggregate dimensions. Evidence dimensionaggregation may be done manually, automatically, or by combination ofboth. Aggregation may include summation, averaging, weighted averaging,or any other mathematical or statistical algorithm that summarizesmultiple dimensions (e.g., of finer granularity) into one aggregateddimension (e.g., coarse level of granularity).

FIGS. 10A-10C illustrate an example QA system, for instance, which mayprovide a statement of hypothesis (or answer), and associated pieces ofsupporting evidence which then further may be profiled according to theevidence profiling disclosed above. The information retrieval system mayidentify a number of relevant sources and further analyze or synthesizethe information contained in those segments to satisfy the user'sinformation need, e.g., to generate the answer or hypothesis. In oneaspect, the QA system may use logical proofs to guide search. A finalconclusion that is produced in response to a query may be onlyindirectly derived from content found in multiple documents. FIG. 10Ashows an example: a question such as “What country is Chicago in?” mightbe answered by separate documents that state that “Chicago is inIllinois” and that “Illinois is in the USA.” Once a system has derived aconclusion (e.g., that Chicago is in the USA), it may be expected toprovide support for that conclusion by citing original sources.

There may be multiple distinct proofs that can be found for any givenconclusion. Each of these proofs may involve some set of assertions thatare directly asserted in source material. Any given assertion may bestated in one or more sources. Some assertions are “leaves” of the prooftree; they occur only in the source material and are not computed fromother assertions. Non-leaf assertions are derived from other assertionsand may be directly stated in the sources. A leaf assertion is supportedby any document that asserts it. A non-leaf assertion is supported byany document that asserts it or by the documents that support for eachof the assertions that it was directly computed from. For example, inFIG. 10B, the assertion locatedIn (Illinois, USA) has two alternativesets of support: states.txt, which directly asserts it, andstateRegions.txt plus regionCountries.txt, which support each of theassertions directly used to compute it.

Some queries have responses that have multiple instances, each with oneor more supporting documents. FIG. 10C illustrates this: “What citiesare in the USA?” has many different answers and some of those answersmay be asserted in a single document while others may emerge only fromthe combination of multiple documents.

At indexing time, given an unstructured (e.g., text) corpus: astructured repository (e.g., knowledge base, search index) may be builtof assertions, e.g., using traditional knowledge extraction technologies(e.g., Indexing). For each assertion, one or more sources (e.g.,document, passage) that asserted it may be recorded. At run time, givena query and a knowledge base, the query may be formalized (referred toas query analysis). Given the formal query, proofs of statements may begenerated that match the query (referred to as Theorem Proving). Foreach proof, a set of documents may be generated (e.g., referred to asDocument Selection). The set of documents may be generated by iteratinga process which, for each assertion A that is a direct antecedent of theconclusion, either chose a source that asserts or iterate over theantecedents of A. The set of documents may be ordered based onrelevance, parsimony, etc. (referred to as Document Set Ranking). In oneembodiment, these steps can be performed using existing technology(e.g., Theorem Proving) and/or with extensions of existing technology(e.g., Document Set Ranking could be performed by extending algorithmsfor ranking individual documents) or others.

FIG. 11 illustrates a schematic of an example computer or processingsystem that may implement the evidence profiling in one embodiment ofthe present disclosure. The computer system is only one example of asuitable processing system and is not intended to suggest any limitationas to the scope of use or functionality of embodiments of themethodology described herein. The processing system shown may beoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with the processing system shown in FIG. 11 mayinclude, but are not limited to, personal computer systems, servercomputer systems, thin clients, thick clients, handheld or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputersystems, mainframe computer systems, and distributed cloud computingenvironments that include any of the above systems or devices, and thelike.

The computer system may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The computer system may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

The components of computer system may include, but are not limited to,one or more processors or processing units 12, a system memory 16, and abus 14 that couples various system components including system memory 16to processor 12. The processor 12 may include an evidence profilingmodule 10 that performs the evidence profiling described herein. Theevidence profiling module 10 may be programmed into the integratedcircuits of the processor 12, or loaded from memory 16, storage device18, or network 24 or combinations thereof.

Bus 14 may represent one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

Computer system may include a variety of computer system readable media.Such media may be any available media that is accessible by computersystem, and it may include both volatile and non-volatile media,removable and non-removable media.

System memory 16 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) and/or cachememory or others. Computer system may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 18 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(e.g., a “hard drive”). Although not shown, a magnetic disk drive forreading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), and an optical disk drive for reading from orwriting to a removable, non-volatile optical disk such as a CD-ROM,DVD-ROM or other optical media can be provided. In such instances, eachcan be connected to bus 14 by one or more data media interfaces.

Computer system may also communicate with one or more external devices26 such as a keyboard, a pointing device, a display 28, etc.; one ormore devices that enable a user to interact with computer system; and/orany devices (e.g., network card, modem, etc.) that enable computersystem to communicate with one or more other computing devices. Suchcommunication can occur via Input/Output (I/O) interfaces 20. Theevidence profiling visualization, for example, may be presented on thedisplay device 28. For instance, the evidence profiling module 10 mayinclude visualization logic for generating the visualization or enablingthe generation of the visualization, which visualization may bepresented on the display device 28.

Still yet, computer system can communicate with one or more networks 24such as a local area network (LAN), a general wide area network (WAN),and/or a public network (e.g., the Internet) via network adapter 22. Asdepicted, network adapter 22 communicates with the other components ofcomputer system via bus 14. It should be understood that although notshown, other hardware and/or software components could be used inconjunction with computer system. Examples, include, but are not limitedto: microcode, device drivers, redundant processing units, external diskdrive arrays, RAID systems, tape drives, and data archival storagesystems, etc.

Evidence profiles provide a way for users to better understand howdifferent kinds of evidence, and different kinds of analyses applied tothat evidence, support or refute one or more hypotheses proposed assolutions to a given problem. For example, the problem may be a naturallanguage question submitted to a Question Answering system, thehypotheses may be candidate answers for that question, and the evidencemay include a wide variety of analyses applied to different kinds ofevidence to support or refute each candidate answer. Evidence profilingmay include receiving a statement of hypothesis generated based oncontent, generating an evidence profile communicating a degree to whichthe content supports the statement of hypothesis as a true statement,and enabling visualization of the evidence profile. An evidence profilemay contain one or more dimensions of evidence which supported thestatement of hypothesis, and a score associated with each of thedimensions of the evidence. Evidence profiling may also includepresenting the evidence profile to the user using a variety ofvisualizations that allow the user to easily understand how a givenhypothesis is supported or refuted by the evidence.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages, a scripting language such as Perl, VBS or similarlanguages, and/or functional languages such as Lisp and ML andlogic-oriented languages such as Prolog. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the present invention are described with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The computer program product may comprise all the respective featuresenabling the implementation of the methodology described herein, andwhich—when loaded in a computer system—is able to carry out the methods.Computer program, software program, program, or software, in the presentcontext means any expression, in any language, code or notation, of aset of instructions intended to cause a system having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: (a) conversion to anotherlanguage, code or notation; and/or (b) reproduction in a differentmaterial form.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements, if any, in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

Various aspects of the present disclosure may be embodied as a program,software, or computer instructions embodied in a computer or machineusable or readable medium, which causes the computer or machine toperform the steps of the method when executed on the computer,processor, and/or machine. A program storage device readable by amachine, tangibly embodying a program of instructions executable by themachine to perform various functionalities and methods described in thepresent disclosure is also provided.

The system and method of the present disclosure may be implemented andrun on a general-purpose computer or special-purpose computer system.The terms “computer system” and “computer network” as may be used in thepresent application may include a variety of combinations of fixedand/or portable computer hardware, software, peripherals, and storagedevices. The computer system may include a plurality of individualcomponents that are networked or otherwise linked to performcollaboratively, or may include one or more stand-alone components. Thehardware and software components of the computer system of the presentapplication may include and may be included within fixed and portabledevices such as desktop, laptop, and/or server. A module may be acomponent of a device, software, program, or system that implements some“functionality”, which can be embodied as software, hardware, firmware,electronic circuitry, or etc.

The embodiments described above are illustrative examples and it shouldnot be construed that the present invention is limited to theseparticular embodiments. Thus, various changes and modifications may beeffected by one skilled in the art without departing from the spirit orscope of the invention as defined in the appended claims.

We claim:
 1. A method for evidence profiling, comprising: receiving acandidate answer and supporting pieces of evidence; and generating, by aprocessor, an evidence profile communicating a degree to which theevidence supports the candidate answer as being correct, wherein theevidence profile provides dimensions of evidence and each dimension maysupport or refute the candidate answer as being correct, wherein thedimensions of evidence comprise at least taxonomic, geospatial,temporal, source reliability, passage support and popularity dimensions,and the supporting pieces of evidence comprise at least content whosefeatures are grouped into said dimensions of evidence used to generatethe evidence profile; presenting one or more candidate answers withassociated confidence levels, the candidate answer being one of said oneor more candidate answers, said one or more candidate answers having acorresponding evidence profile, wherein a user is enabled to navigate tothe evidence profile by selecting said one or more candidate answers,wherein a visualization of the evidence profile presents said selectedone or more candidate answers by the dimensions with associateddimension scores, said dimension scores representing degrees to whichthe supporting pieces of evidence corresponding to the dimensionssupport said selected one or more candidate answers as being correct,wherein the user is enabled to further navigate to the supporting piecesof evidence used to produce the dimension scores via the visualization.2. The method of claim 1 further comprising configuring thevisualization of the evidence profile such that each dimension ofevidence can be selected and a weight associated with each dimensionaltered.
 3. The method of claim 1, further comprising: configuring aselection of a desired dimension; and displaying pieces of evidencecontributing to a score for the selected dimension.
 4. The method ofclaim 1, wherein the generating an evidence profile includes:determining dimensions of evidence based on types of algorithms thatwere used to gather and score each piece of supporting evidence; andobtaining a score associated with each of the dimensions of the evidenceby aggregating the scores for each piece of supporting evidence.
 5. Themethod of claim 3, wherein the dimensions of evidence are aggregatedinto a hierarchy.
 6. The method of claim 5, wherein the dimensions ofevidence are aggregated automatically or manually or combinationsthereof.
 7. The method of claim 2, wherein the visualization includespresenting relative contributions of each of the dimensions of evidencesupporting the candidate answer.
 8. The method of claim 2, wherein thevisualization includes showing comparison of two or more candidateanswers, in which the dimensions of evidence supporting said two or morecandidate answers are shown.
 9. The method of claim 8, wherein only thedimensions of evidence that provide different levels of support for thetwo or more candidate answers are shown.
 10. The method of claim 9,wherein relative differences in contribution of the dimensions ofevidence between the two or more candidate answers are shown.